{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cache entry deserialization failed, entry ignored\n",
      "mxnet 1.5.1.post0 has requirement numpy<2.0.0,>1.16.0, but you'll have numpy 1.14.3 which is incompatible.\n",
      "mizani 0.6.0 has requirement matplotlib>=3.1.1, but you'll have matplotlib 3.0.3 which is incompatible.\n",
      "mizani 0.6.0 has requirement pandas>=0.25.0, but you'll have pandas 0.23.0 which is incompatible.\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "pip -q install --upgrade pip\n",
    "pip -q install sagemaker awscli boto3 --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>Jupyter.notebook.kernel.restart()</script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "HTML(\"<script>Jupyter.notebook.kernel.restart()</script>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Direct Marketing with Amazon SageMaker XGBoost and Hyperparameter Tuning\n",
    "_**Supervised Learning with Gradient Boosted Trees: A Binary Prediction Problem With Unbalanced Classes**_\n",
    "\n",
    "Last update: December 2nd, 2019\n",
    "\n",
    "---\n",
    "\n",
    "## Background\n",
    "Direct marketing, either through mail, email, phone, etc., is a common tactic to acquire customers.  Because resources and a customer's attention is limited, the goal is to only target the subset of prospects who are likely to engage with a specific offer.  Predicting those potential customers based on readily available information like demographics, past interactions, and environmental factors is a common machine learning problem.\n",
    "\n",
    "This notebook will train a model which can be used to predict if a customer will enroll for a term deposit at a bank, after one or more phone calls. Hyperparameter tuning will be used in order to try multiple hyperparameter settings and produce the best model.\n",
    "\n",
    "We will use SageMaker Python SDK, a high level SDK, to simplify the way we interact with SageMaker Hyperparameter Tuning.\n",
    "\n",
    "---\n",
    "\n",
    "## Preparation\n",
    "\n",
    "Let's start by specifying:\n",
    "\n",
    "- The S3 bucket and prefix that you want to use for training and model data.  We'll use the default bucket. If you want to use your own, make sure it is in the **same region** as SageMaker.\n",
    "- The IAM role used to give training access to your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "isConfigCell": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.50.8\n"
     ]
    }
   ],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "import os \n",
    " \n",
    "print (sagemaker.__version__)\n",
    "\n",
    "bucket = sagemaker.Session().default_bucket()                     \n",
    "prefix = 'sagemaker-autopilot/DEMO-hpo-xgboost-dm'\n",
    "\n",
    "# Role when working on a notebook instance\n",
    "role = sagemaker.get_execution_role()\n",
    "# Role when working locally\n",
    "# role = ROLE_ARN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading the data set\n",
    "Let's start by downloading the [direct marketing dataset](https://archive.ics.uci.edu/ml/datasets/bank+marketing) from UCI's ML Repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-02-04 17:46:28--  https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank-additional.zip\n",
      "Resolving archive.ics.uci.edu (archive.ics.uci.edu)... 128.195.10.252\n",
      "Connecting to archive.ics.uci.edu (archive.ics.uci.edu)|128.195.10.252|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 444572 (434K) [application/x-httpd-php]\n",
      "Saving to: ‘bank-additional.zip’\n",
      "\n",
      "bank-additional.zip 100%[===================>] 434.15K  1.57MB/s    in 0.3s    \n",
      "\n",
      "2020-02-04 17:46:29 (1.57 MB/s) - ‘bank-additional.zip’ saved [444572/444572]\n",
      "\n",
      "Archive:  bank-additional.zip\n",
      "   creating: bank-additional/\n",
      "  inflating: bank-additional/.DS_Store  \n",
      "   creating: __MACOSX/\n",
      "   creating: __MACOSX/bank-additional/\n",
      "  inflating: __MACOSX/bank-additional/._.DS_Store  \n",
      "  inflating: bank-additional/.Rhistory  \n",
      "  inflating: bank-additional/bank-additional-full.csv  \n",
      "  inflating: bank-additional/bank-additional-names.txt  \n",
      "  inflating: bank-additional/bank-additional.csv  \n",
      "  inflating: __MACOSX/._bank-additional  \n"
     ]
    }
   ],
   "source": [
    "!wget -N https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank-additional.zip\n",
    "!unzip -o bank-additional.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"age\";\"job\";\"marital\";\"education\";\"default\";\"housing\";\"loan\";\"contact\";\"month\";\"day_of_week\";\"duration\";\"campaign\";\"pdays\";\"previous\";\"poutcome\";\"emp.var.rate\";\"cons.price.idx\";\"cons.conf.idx\";\"euribor3m\";\"nr.employed\";\"y\"\n",
      "56;\"housemaid\";\"married\";\"basic.4y\";\"no\";\"no\";\"no\";\"telephone\";\"may\";\"mon\";261;1;999;0;\"nonexistent\";1.1;93.994;-36.4;4.857;5191;\"no\"\n",
      "57;\"services\";\"married\";\"high.school\";\"unknown\";\"no\";\"no\";\"telephone\";\"may\";\"mon\";149;1;999;0;\"nonexistent\";1.1;93.994;-36.4;4.857;5191;\"no\"\n",
      "37;\"services\";\"married\";\"high.school\";\"no\";\"yes\";\"no\";\"telephone\";\"may\";\"mon\";226;1;999;0;\"nonexistent\";1.1;93.994;-36.4;4.857;5191;\"no\"\n",
      "40;\"admin.\";\"married\";\"basic.6y\";\"no\";\"no\";\"no\";\"telephone\";\"may\";\"mon\";151;1;999;0;\"nonexistent\";1.1;93.994;-36.4;4.857;5191;\"no\"\n",
      "56;\"services\";\"married\";\"high.school\";\"no\";\"no\";\"yes\";\"telephone\";\"may\";\"mon\";307;1;999;0;\"nonexistent\";1.1;93.994;-36.4;4.857;5191;\"no\"\n",
      "45;\"services\";\"married\";\"basic.9y\";\"unknown\";\"no\";\"no\";\"telephone\";\"may\";\"mon\";198;1;999;0;\"nonexistent\";1.1;93.994;-36.4;4.857;5191;\"no\"\n",
      "59;\"admin.\";\"married\";\"professional.course\";\"no\";\"no\";\"no\";\"telephone\";\"may\";\"mon\";139;1;999;0;\"nonexistent\";1.1;93.994;-36.4;4.857;5191;\"no\"\n",
      "41;\"blue-collar\";\"married\";\"unknown\";\"unknown\";\"no\";\"no\";\"telephone\";\"may\";\"mon\";217;1;999;0;\"nonexistent\";1.1;93.994;-36.4;4.857;5191;\"no\"\n",
      "24;\"technician\";\"single\";\"professional.course\";\"no\";\"yes\";\"no\";\"telephone\";\"may\";\"mon\";380;1;999;0;\"nonexistent\";1.1;93.994;-36.4;4.857;5191;\"no\"\n"
     ]
    }
   ],
   "source": [
    "!head ./bank-additional/bank-additional-full.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to load this CSV file, inspect it, pre-process it, etc. Please don't write custom Python code to do this!\n",
    "\n",
    "Instead, developers typically use libraries such as:\n",
    "* [Pandas](https://pandas.pydata.org/): a library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.\n",
    "* [Numpy](http://www.numpy.org/): a fundamental package for scientific computing with Python.\n",
    "\n",
    "Along the way, we'll use functions from these two libraries. You should definitely become familiar with them, they will make your life much easier when working with large datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np  # For matrix operations and numerical processing\n",
    "import pandas as pd # For munging tabular data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read the CSV file into a Pandas data frame and take a look at the first few lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>job</th>\n",
       "      <th>marital</th>\n",
       "      <th>education</th>\n",
       "      <th>default</th>\n",
       "      <th>housing</th>\n",
       "      <th>loan</th>\n",
       "      <th>contact</th>\n",
       "      <th>month</th>\n",
       "      <th>day_of_week</th>\n",
       "      <th>duration</th>\n",
       "      <th>campaign</th>\n",
       "      <th>pdays</th>\n",
       "      <th>previous</th>\n",
       "      <th>poutcome</th>\n",
       "      <th>emp.var.rate</th>\n",
       "      <th>cons.price.idx</th>\n",
       "      <th>cons.conf.idx</th>\n",
       "      <th>euribor3m</th>\n",
       "      <th>nr.employed</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>56</td>\n",
       "      <td>housemaid</td>\n",
       "      <td>married</td>\n",
       "      <td>basic.4y</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>telephone</td>\n",
       "      <td>may</td>\n",
       "      <td>mon</td>\n",
       "      <td>261</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>57</td>\n",
       "      <td>services</td>\n",
       "      <td>married</td>\n",
       "      <td>high.school</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>telephone</td>\n",
       "      <td>may</td>\n",
       "      <td>mon</td>\n",
       "      <td>149</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37</td>\n",
       "      <td>services</td>\n",
       "      <td>married</td>\n",
       "      <td>high.school</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>telephone</td>\n",
       "      <td>may</td>\n",
       "      <td>mon</td>\n",
       "      <td>226</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>40</td>\n",
       "      <td>admin.</td>\n",
       "      <td>married</td>\n",
       "      <td>basic.6y</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>telephone</td>\n",
       "      <td>may</td>\n",
       "      <td>mon</td>\n",
       "      <td>151</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>56</td>\n",
       "      <td>services</td>\n",
       "      <td>married</td>\n",
       "      <td>high.school</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>telephone</td>\n",
       "      <td>may</td>\n",
       "      <td>mon</td>\n",
       "      <td>307</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>45</td>\n",
       "      <td>services</td>\n",
       "      <td>married</td>\n",
       "      <td>basic.9y</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>telephone</td>\n",
       "      <td>may</td>\n",
       "      <td>mon</td>\n",
       "      <td>198</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>59</td>\n",
       "      <td>admin.</td>\n",
       "      <td>married</td>\n",
       "      <td>professional.course</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>telephone</td>\n",
       "      <td>may</td>\n",
       "      <td>mon</td>\n",
       "      <td>139</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>41</td>\n",
       "      <td>blue-collar</td>\n",
       "      <td>married</td>\n",
       "      <td>unknown</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>telephone</td>\n",
       "      <td>may</td>\n",
       "      <td>mon</td>\n",
       "      <td>217</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>24</td>\n",
       "      <td>technician</td>\n",
       "      <td>single</td>\n",
       "      <td>professional.course</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>telephone</td>\n",
       "      <td>may</td>\n",
       "      <td>mon</td>\n",
       "      <td>380</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>25</td>\n",
       "      <td>services</td>\n",
       "      <td>single</td>\n",
       "      <td>high.school</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>telephone</td>\n",
       "      <td>may</td>\n",
       "      <td>mon</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age          job  marital            education  default housing loan  \\\n",
       "0   56    housemaid  married             basic.4y       no      no   no   \n",
       "1   57     services  married          high.school  unknown      no   no   \n",
       "2   37     services  married          high.school       no     yes   no   \n",
       "3   40       admin.  married             basic.6y       no      no   no   \n",
       "4   56     services  married          high.school       no      no  yes   \n",
       "5   45     services  married             basic.9y  unknown      no   no   \n",
       "6   59       admin.  married  professional.course       no      no   no   \n",
       "7   41  blue-collar  married              unknown  unknown      no   no   \n",
       "8   24   technician   single  professional.course       no     yes   no   \n",
       "9   25     services   single          high.school       no     yes   no   \n",
       "\n",
       "     contact month day_of_week  duration  campaign  pdays  previous  \\\n",
       "0  telephone   may         mon       261         1    999         0   \n",
       "1  telephone   may         mon       149         1    999         0   \n",
       "2  telephone   may         mon       226         1    999         0   \n",
       "3  telephone   may         mon       151         1    999         0   \n",
       "4  telephone   may         mon       307         1    999         0   \n",
       "5  telephone   may         mon       198         1    999         0   \n",
       "6  telephone   may         mon       139         1    999         0   \n",
       "7  telephone   may         mon       217         1    999         0   \n",
       "8  telephone   may         mon       380         1    999         0   \n",
       "9  telephone   may         mon        50         1    999         0   \n",
       "\n",
       "      poutcome  emp.var.rate  cons.price.idx  cons.conf.idx  euribor3m  \\\n",
       "0  nonexistent           1.1          93.994          -36.4      4.857   \n",
       "1  nonexistent           1.1          93.994          -36.4      4.857   \n",
       "2  nonexistent           1.1          93.994          -36.4      4.857   \n",
       "3  nonexistent           1.1          93.994          -36.4      4.857   \n",
       "4  nonexistent           1.1          93.994          -36.4      4.857   \n",
       "5  nonexistent           1.1          93.994          -36.4      4.857   \n",
       "6  nonexistent           1.1          93.994          -36.4      4.857   \n",
       "7  nonexistent           1.1          93.994          -36.4      4.857   \n",
       "8  nonexistent           1.1          93.994          -36.4      4.857   \n",
       "9  nonexistent           1.1          93.994          -36.4      4.857   \n",
       "\n",
       "   nr.employed   y  \n",
       "0       5191.0  no  \n",
       "1       5191.0  no  \n",
       "2       5191.0  no  \n",
       "3       5191.0  no  \n",
       "4       5191.0  no  \n",
       "5       5191.0  no  \n",
       "6       5191.0  no  \n",
       "7       5191.0  no  \n",
       "8       5191.0  no  \n",
       "9       5191.0  no  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html\n",
    "data = pd.read_csv('./bank-additional/bank-additional-full.csv', sep=';')\n",
    "pd.set_option('display.max_columns', 500)     # Make sure we can see all of the columns\n",
    "pd.set_option('display.max_rows', 50)         # Keep the output on one page\n",
    "data[:10] # Show the first 10 lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(41188, 21)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape # (number of lines, number of columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two classes are extremely unbalanced and it could be a problem for our classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Positive samples: 4640\n",
      "Negative samples: 36548\n",
      "Ratio: 7.88\n"
     ]
    }
   ],
   "source": [
    "one_class = data[data['y']=='yes']\n",
    "one_class_count = one_class.shape[0]\n",
    "print(\"Positive samples: %d\" % one_class_count)\n",
    "\n",
    "zero_class = data[data['y']=='no']\n",
    "zero_class_count = zero_class.shape[0]\n",
    "print(\"Negative samples: %d\" % zero_class_count)\n",
    "\n",
    "zero_to_one_ratio = zero_class_count/one_class_count\n",
    "print(\"Ratio: %.2f\" % zero_to_one_ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's talk about the data.  At a high level, we can see:\n",
    "\n",
    "* We have a little over 40K customer records, 20 features plus a target variable ('y') for each customer\n",
    "* The features are mixed; some numeric, some categorical\n",
    "* The data appears to be sorted, at least by `time` and `contact`, maybe more\n",
    "\n",
    "_**Specifics on each of the features:**_\n",
    "\n",
    "*Demographics:*\n",
    "* `age`: Customer's age (numeric)\n",
    "* `job`: Type of job (categorical: 'admin.', 'services', ...)\n",
    "* `marital`: Marital status (categorical: 'married', 'single', ...)\n",
    "* `education`: Level of education (categorical: 'basic.4y', 'high.school', ...)\n",
    "\n",
    "*Past customer events:*\n",
    "* `default`: Has credit in default? (categorical: 'no', 'unknown', ...)\n",
    "* `housing`: Has housing loan? (categorical: 'no', 'yes', ...)\n",
    "* `loan`: Has personal loan? (categorical: 'no', 'yes', ...)\n",
    "\n",
    "*Past direct marketing contacts:*\n",
    "* `contact`: Contact communication type (categorical: 'cellular', 'telephone', ...)\n",
    "* `month`: Last contact month of year (categorical: 'may', 'nov', ...)\n",
    "* `day_of_week`: Last contact day of the week (categorical: 'mon', 'fri', ...)\n",
    "* `duration`: Last contact duration, in seconds (numeric). Important note: If duration = 0 then `y` = 'no'.\n",
    " \n",
    "*Campaign information:*\n",
    "* `campaign`: Number of contacts performed during this campaign and for this client (numeric, includes last contact)\n",
    "* `pdays`: Number of days that passed by after the client was last contacted from a previous campaign (numeric)\n",
    "* `previous`: Number of contacts performed before this campaign and for this client (numeric)\n",
    "* `poutcome`: Outcome of the previous marketing campaign (categorical: 'nonexistent','success', ...)\n",
    "\n",
    "*External environment factors:*\n",
    "* `emp.var.rate`: Employment variation rate - quarterly indicator (numeric)\n",
    "* `cons.price.idx`: Consumer price index - monthly indicator (numeric)\n",
    "* `cons.conf.idx`: Consumer confidence index - monthly indicator (numeric)\n",
    "* `euribor3m`: Euribor 3 month rate - daily indicator (numeric)\n",
    "* `nr.employed`: Number of employees - quarterly indicator (numeric)\n",
    "\n",
    "*Target variable:*\n",
    "* `y`: Has the client subscribed a term deposit? (binary: 'yes','no')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transforming the dataset\n",
    "Cleaning up data is part of nearly every machine learning project.  It arguably presents the biggest risk if done incorrectly and is one of the more subjective aspects in the process.  Several common techniques include:\n",
    "\n",
    "* Handling missing values: Some machine learning algorithms are capable of handling missing values, but most would rather not.  Options include:\n",
    " * Removing observations with missing values: This works well if only a very small fraction of observations have incomplete information.\n",
    " * Removing features with missing values: This works well if there are a small number of features which have a large number of missing values.\n",
    " * Imputing missing values: Entire [books](https://www.amazon.com/Flexible-Imputation-Missing-Interdisciplinary-Statistics/dp/1439868247) have been written on this topic, but common choices are replacing the missing value with the mode or mean of that column's non-missing values.\n",
    "* Converting categorical to numeric: The most common method is one hot encoding, which for each feature maps every distinct value of that column to its own feature which takes a value of 1 when the categorical feature is equal to that value, and 0 otherwise.\n",
    "* Oddly distributed data: Although for non-linear models like Gradient Boosted Trees, this has very limited implications, parametric models like regression can produce wildly inaccurate estimates when fed highly skewed data.  In some cases, simply taking the natural log of the features is sufficient to produce more normally distributed data.  In others, bucketing values into discrete ranges is helpful.  These buckets can then be treated as categorical variables and included in the model when one hot encoded.\n",
    "* Handling more complicated data types: Mainpulating images, text, or data at varying grains.\n",
    "\n",
    "Luckily, some of these aspects have already been handled for us, and the algorithm we are showcasing tends to do well at handling sparse or oddly distributed data.  Therefore, let's keep pre-processing simple."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, many records have the value of \"999\" for pdays, number of days that passed by after a client was last contacted. It is very likely to be a magic number to represent that no contact was made before. Considering that, we create a new column called \"no_previous_contact\", then grant it value of \"1\" when pdays is 999 and \"0\" otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 999]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[np.min(data['pdays']), np.max(data['pdays'])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Indicator variable to capture when pdays takes a value of 999\n",
    "# https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.where.html\n",
    "data['no_previous_contact'] = np.where(data['pdays'] == 999, 1, 0)\n",
    "data = data.drop(['pdays'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the \"job\" column, there are categories that mean the customer is not working, e.g., \"student\", \"retire\", and \"unemployed\". Since it is very likely whether or not a customer is working will affect his/her decision to enroll in the term deposit, we generate a new column to show whether the customer is working based on \"job\" column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "admin.           10422\n",
       "blue-collar       9254\n",
       "technician        6743\n",
       "services          3969\n",
       "management        2924\n",
       "retired           1720\n",
       "entrepreneur      1456\n",
       "self-employed     1421\n",
       "housemaid         1060\n",
       "unemployed        1014\n",
       "student            875\n",
       "unknown            330\n",
       "Name: job, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['job'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Indicator for individuals not actively employed\n",
    "# https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.in1d.html\n",
    "data['not_working'] = np.where(np.in1d(data['job'], ['student', 'retired', 'unemployed']), 1, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last but not the least, we convert categorical to numeric, as is suggested above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>duration</th>\n",
       "      <th>campaign</th>\n",
       "      <th>previous</th>\n",
       "      <th>emp.var.rate</th>\n",
       "      <th>cons.price.idx</th>\n",
       "      <th>cons.conf.idx</th>\n",
       "      <th>euribor3m</th>\n",
       "      <th>nr.employed</th>\n",
       "      <th>no_previous_contact</th>\n",
       "      <th>not_working</th>\n",
       "      <th>job_admin.</th>\n",
       "      <th>job_blue-collar</th>\n",
       "      <th>job_entrepreneur</th>\n",
       "      <th>job_housemaid</th>\n",
       "      <th>job_management</th>\n",
       "      <th>job_retired</th>\n",
       "      <th>job_self-employed</th>\n",
       "      <th>job_services</th>\n",
       "      <th>job_student</th>\n",
       "      <th>job_technician</th>\n",
       "      <th>job_unemployed</th>\n",
       "      <th>job_unknown</th>\n",
       "      <th>marital_divorced</th>\n",
       "      <th>marital_married</th>\n",
       "      <th>marital_single</th>\n",
       "      <th>marital_unknown</th>\n",
       "      <th>education_basic.4y</th>\n",
       "      <th>education_basic.6y</th>\n",
       "      <th>education_basic.9y</th>\n",
       "      <th>education_high.school</th>\n",
       "      <th>education_illiterate</th>\n",
       "      <th>education_professional.course</th>\n",
       "      <th>education_university.degree</th>\n",
       "      <th>education_unknown</th>\n",
       "      <th>default_no</th>\n",
       "      <th>default_unknown</th>\n",
       "      <th>default_yes</th>\n",
       "      <th>housing_no</th>\n",
       "      <th>housing_unknown</th>\n",
       "      <th>housing_yes</th>\n",
       "      <th>loan_no</th>\n",
       "      <th>loan_unknown</th>\n",
       "      <th>loan_yes</th>\n",
       "      <th>contact_cellular</th>\n",
       "      <th>contact_telephone</th>\n",
       "      <th>month_apr</th>\n",
       "      <th>month_aug</th>\n",
       "      <th>month_dec</th>\n",
       "      <th>month_jul</th>\n",
       "      <th>month_jun</th>\n",
       "      <th>month_mar</th>\n",
       "      <th>month_may</th>\n",
       "      <th>month_nov</th>\n",
       "      <th>month_oct</th>\n",
       "      <th>month_sep</th>\n",
       "      <th>day_of_week_fri</th>\n",
       "      <th>day_of_week_mon</th>\n",
       "      <th>day_of_week_thu</th>\n",
       "      <th>day_of_week_tue</th>\n",
       "      <th>day_of_week_wed</th>\n",
       "      <th>poutcome_failure</th>\n",
       "      <th>poutcome_nonexistent</th>\n",
       "      <th>poutcome_success</th>\n",
       "      <th>y_no</th>\n",
       "      <th>y_yes</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>56</td>\n",
       "      <td>261</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>57</td>\n",
       "      <td>149</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37</td>\n",
       "      <td>226</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>40</td>\n",
       "      <td>151</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>56</td>\n",
       "      <td>307</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>45</td>\n",
       "      <td>198</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>59</td>\n",
       "      <td>139</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>41</td>\n",
       "      <td>217</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>24</td>\n",
       "      <td>380</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>25</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.857</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  duration  campaign  previous  emp.var.rate  cons.price.idx  \\\n",
       "0   56       261         1         0           1.1          93.994   \n",
       "1   57       149         1         0           1.1          93.994   \n",
       "2   37       226         1         0           1.1          93.994   \n",
       "3   40       151         1         0           1.1          93.994   \n",
       "4   56       307         1         0           1.1          93.994   \n",
       "5   45       198         1         0           1.1          93.994   \n",
       "6   59       139         1         0           1.1          93.994   \n",
       "7   41       217         1         0           1.1          93.994   \n",
       "8   24       380         1         0           1.1          93.994   \n",
       "9   25        50         1         0           1.1          93.994   \n",
       "\n",
       "   cons.conf.idx  euribor3m  nr.employed  no_previous_contact  not_working  \\\n",
       "0          -36.4      4.857       5191.0                    1            0   \n",
       "1          -36.4      4.857       5191.0                    1            0   \n",
       "2          -36.4      4.857       5191.0                    1            0   \n",
       "3          -36.4      4.857       5191.0                    1            0   \n",
       "4          -36.4      4.857       5191.0                    1            0   \n",
       "5          -36.4      4.857       5191.0                    1            0   \n",
       "6          -36.4      4.857       5191.0                    1            0   \n",
       "7          -36.4      4.857       5191.0                    1            0   \n",
       "8          -36.4      4.857       5191.0                    1            0   \n",
       "9          -36.4      4.857       5191.0                    1            0   \n",
       "\n",
       "   job_admin.  job_blue-collar  job_entrepreneur  job_housemaid  \\\n",
       "0           0                0                 0              1   \n",
       "1           0                0                 0              0   \n",
       "2           0                0                 0              0   \n",
       "3           1                0                 0              0   \n",
       "4           0                0                 0              0   \n",
       "5           0                0                 0              0   \n",
       "6           1                0                 0              0   \n",
       "7           0                1                 0              0   \n",
       "8           0                0                 0              0   \n",
       "9           0                0                 0              0   \n",
       "\n",
       "   job_management  job_retired  job_self-employed  job_services  job_student  \\\n",
       "0               0            0                  0             0            0   \n",
       "1               0            0                  0             1            0   \n",
       "2               0            0                  0             1            0   \n",
       "3               0            0                  0             0            0   \n",
       "4               0            0                  0             1            0   \n",
       "5               0            0                  0             1            0   \n",
       "6               0            0                  0             0            0   \n",
       "7               0            0                  0             0            0   \n",
       "8               0            0                  0             0            0   \n",
       "9               0            0                  0             1            0   \n",
       "\n",
       "   job_technician  job_unemployed  job_unknown  marital_divorced  \\\n",
       "0               0               0            0                 0   \n",
       "1               0               0            0                 0   \n",
       "2               0               0            0                 0   \n",
       "3               0               0            0                 0   \n",
       "4               0               0            0                 0   \n",
       "5               0               0            0                 0   \n",
       "6               0               0            0                 0   \n",
       "7               0               0            0                 0   \n",
       "8               1               0            0                 0   \n",
       "9               0               0            0                 0   \n",
       "\n",
       "   marital_married  marital_single  marital_unknown  education_basic.4y  \\\n",
       "0                1               0                0                   1   \n",
       "1                1               0                0                   0   \n",
       "2                1               0                0                   0   \n",
       "3                1               0                0                   0   \n",
       "4                1               0                0                   0   \n",
       "5                1               0                0                   0   \n",
       "6                1               0                0                   0   \n",
       "7                1               0                0                   0   \n",
       "8                0               1                0                   0   \n",
       "9                0               1                0                   0   \n",
       "\n",
       "   education_basic.6y  education_basic.9y  education_high.school  \\\n",
       "0                   0                   0                      0   \n",
       "1                   0                   0                      1   \n",
       "2                   0                   0                      1   \n",
       "3                   1                   0                      0   \n",
       "4                   0                   0                      1   \n",
       "5                   0                   1                      0   \n",
       "6                   0                   0                      0   \n",
       "7                   0                   0                      0   \n",
       "8                   0                   0                      0   \n",
       "9                   0                   0                      1   \n",
       "\n",
       "   education_illiterate  education_professional.course  \\\n",
       "0                     0                              0   \n",
       "1                     0                              0   \n",
       "2                     0                              0   \n",
       "3                     0                              0   \n",
       "4                     0                              0   \n",
       "5                     0                              0   \n",
       "6                     0                              1   \n",
       "7                     0                              0   \n",
       "8                     0                              1   \n",
       "9                     0                              0   \n",
       "\n",
       "   education_university.degree  education_unknown  default_no  \\\n",
       "0                            0                  0           1   \n",
       "1                            0                  0           0   \n",
       "2                            0                  0           1   \n",
       "3                            0                  0           1   \n",
       "4                            0                  0           1   \n",
       "5                            0                  0           0   \n",
       "6                            0                  0           1   \n",
       "7                            0                  1           0   \n",
       "8                            0                  0           1   \n",
       "9                            0                  0           1   \n",
       "\n",
       "   default_unknown  default_yes  housing_no  housing_unknown  housing_yes  \\\n",
       "0                0            0           1                0            0   \n",
       "1                1            0           1                0            0   \n",
       "2                0            0           0                0            1   \n",
       "3                0            0           1                0            0   \n",
       "4                0            0           1                0            0   \n",
       "5                1            0           1                0            0   \n",
       "6                0            0           1                0            0   \n",
       "7                1            0           1                0            0   \n",
       "8                0            0           0                0            1   \n",
       "9                0            0           0                0            1   \n",
       "\n",
       "   loan_no  loan_unknown  loan_yes  contact_cellular  contact_telephone  \\\n",
       "0        1             0         0                 0                  1   \n",
       "1        1             0         0                 0                  1   \n",
       "2        1             0         0                 0                  1   \n",
       "3        1             0         0                 0                  1   \n",
       "4        0             0         1                 0                  1   \n",
       "5        1             0         0                 0                  1   \n",
       "6        1             0         0                 0                  1   \n",
       "7        1             0         0                 0                  1   \n",
       "8        1             0         0                 0                  1   \n",
       "9        1             0         0                 0                  1   \n",
       "\n",
       "   month_apr  month_aug  month_dec  month_jul  month_jun  month_mar  \\\n",
       "0          0          0          0          0          0          0   \n",
       "1          0          0          0          0          0          0   \n",
       "2          0          0          0          0          0          0   \n",
       "3          0          0          0          0          0          0   \n",
       "4          0          0          0          0          0          0   \n",
       "5          0          0          0          0          0          0   \n",
       "6          0          0          0          0          0          0   \n",
       "7          0          0          0          0          0          0   \n",
       "8          0          0          0          0          0          0   \n",
       "9          0          0          0          0          0          0   \n",
       "\n",
       "   month_may  month_nov  month_oct  month_sep  day_of_week_fri  \\\n",
       "0          1          0          0          0                0   \n",
       "1          1          0          0          0                0   \n",
       "2          1          0          0          0                0   \n",
       "3          1          0          0          0                0   \n",
       "4          1          0          0          0                0   \n",
       "5          1          0          0          0                0   \n",
       "6          1          0          0          0                0   \n",
       "7          1          0          0          0                0   \n",
       "8          1          0          0          0                0   \n",
       "9          1          0          0          0                0   \n",
       "\n",
       "   day_of_week_mon  day_of_week_thu  day_of_week_tue  day_of_week_wed  \\\n",
       "0                1                0                0                0   \n",
       "1                1                0                0                0   \n",
       "2                1                0                0                0   \n",
       "3                1                0                0                0   \n",
       "4                1                0                0                0   \n",
       "5                1                0                0                0   \n",
       "6                1                0                0                0   \n",
       "7                1                0                0                0   \n",
       "8                1                0                0                0   \n",
       "9                1                0                0                0   \n",
       "\n",
       "   poutcome_failure  poutcome_nonexistent  poutcome_success  y_no  y_yes  \n",
       "0                 0                     1                 0     1      0  \n",
       "1                 0                     1                 0     1      0  \n",
       "2                 0                     1                 0     1      0  \n",
       "3                 0                     1                 0     1      0  \n",
       "4                 0                     1                 0     1      0  \n",
       "5                 0                     1                 0     1      0  \n",
       "6                 0                     1                 0     1      0  \n",
       "7                 0                     1                 0     1      0  \n",
       "8                 0                     1                 0     1      0  \n",
       "9                 0                     1                 0     1      0  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html\n",
    "model_data = pd.get_dummies(data)  # Convert categorical variables to sets of indicators\n",
    "model_data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, each categorical column (job, marital, education, etc.) has been replaced by a set of new columns, one for each possible value in the category. Accordingly, we now have 67 columns instead of 21."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(41188, 66)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting features\n",
    "\n",
    "Another question to ask yourself before building a model is whether certain features will add value in your final use case.  For example, if your goal is to deliver the best prediction, then will you have access to that data at the moment of prediction?  Knowing it's raining is highly predictive for umbrella sales, but forecasting weather far enough out to plan inventory on umbrellas is probably just as difficult as forecasting umbrella sales without knowledge of the weather.  So, including this in your model may give you a false sense of precision.\n",
    "\n",
    "Following this logic, let's remove the economic features and `duration` from our data as they would need to be forecasted with high precision to use as inputs in future predictions.\n",
    "\n",
    "Even if we were to use values of the economic indicators from the previous quarter, this value is likely not as relevant for prospects contacted early in the next quarter as those contacted later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop.html\n",
    "model_data = model_data.drop(['duration', 'emp.var.rate', 'cons.price.idx', 'cons.conf.idx', 'euribor3m', 'nr.employed'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(41188, 60)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting the dataset\n",
    "\n",
    "We'll then split the dataset into training (70%), validation (20%), and test (10%) datasets and convert the datasets to the right format the algorithm expects. We will use training and validation datasets during training and we will try to maximize the accuracy on the validation dataset.\n",
    " \n",
    "Once the model has been deployed, we'll use the test dataset to evaluate its performance.\n",
    "\n",
    "Amazon SageMaker's XGBoost algorithm expects data in the libSVM or CSV data format.  For this example, we'll stick to CSV.  Note that the first column must be the target variable and the CSV should not include headers.  Also, notice that although repetitive it's easiest to do this after the train|validation|test split rather than before.  This avoids any misalignment issues due to random reordering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the seed to 123 for reproductibility\n",
    "# https://pandas.pydata.org/pandas-docs/version/0.25/generated/pandas.DataFrame.sample.html\n",
    "# https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.split.html\n",
    "train_data, validation_data, test_data = np.split(model_data.sample(frac=1, random_state=123), \n",
    "                                                  [int(0.7 * len(model_data)), int(0.9*len(model_data))])  \n",
    "\n",
    "# Drop the two columns for 'yes' and 'no' and add 'yes' back as first column of the dataframe\n",
    "# https://pandas.pydata.org/pandas-docs/stable/generated/pandas.concat.html\n",
    "pd.concat([train_data['y_yes'], train_data.drop(['y_no', 'y_yes'], axis=1)], axis=1).to_csv('train.csv', index=False, header=False)\n",
    "pd.concat([validation_data['y_yes'], validation_data.drop(['y_no', 'y_yes'], axis=1)], axis=1).to_csv('validation.csv', index=False, header=False)\n",
    "pd.concat([test_data['y_yes'], test_data.drop(['y_no', 'y_yes'], axis=1)], axis=1).to_csv('test.csv', index=False, header=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-r-- 1 ec2-user ec2-user  490260 Feb  4 17:47 test.csv\n",
      "-rw-rw-r-- 1 ec2-user ec2-user 3431668 Feb  4 17:47 train.csv\n",
      "-rw-rw-r-- 1 ec2-user ec2-user  980538 Feb  4 17:47 validation.csv\n"
     ]
    }
   ],
   "source": [
    "!ls -l *.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll copy the files to S3 for Amazon SageMaker training to pickup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'train/train.csv')).upload_file('train.csv')\n",
    "boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'validation/validation.csv')).upload_file('validation.csv')\n",
    "boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'test/test.csv')).upload_file('test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SageMaker needs to know where the training and validation sets are located, so let's define that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_input_train = sagemaker.s3_input(s3_data='s3://{}/{}/train'.format(bucket, prefix), content_type='csv')\n",
    "s3_input_validation = sagemaker.s3_input(s3_data='s3://{}/{}/validation/'.format(bucket, prefix), content_type='csv')\n",
    "\n",
    "s3_data = {'train': s3_input_train, 'validation': s3_input_validation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training our model\n",
    "\n",
    "The problem we're trying to solve is a classification problem: will a given customer react positively to our marketing offer or not? In order to answer this question, let's train a classification model with XGBoost, a popular open source project available in SageMaker.\n",
    "\n",
    "Please take a few minutes to read:\n",
    "* [The XGBoost algorithm documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html) \n",
    "* [The Estimator API documentation](https://sagemaker.readthedocs.io/en/stable/estimators.html): The Estimator object is central to training activities in SageMaker, **you should be very familiar with it**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "sess = sagemaker.Session()\n",
    "\n",
    "region = boto3.Session().region_name    \n",
    "container = get_image_uri(region, 'xgboost', repo_version='0.90-1')\n",
    "\n",
    "xgb = sagemaker.estimator.Estimator(container,\n",
    "                                    role, \n",
    "                                    train_instance_count=1, \n",
    "                                    train_instance_type='ml.m4.xlarge',\n",
    "                                    input_mode=\"File\",\n",
    "                                    output_path='s3://{}/{}/output'.format(bucket, prefix),\n",
    "                                    sagemaker_session=sess,\n",
    "                                    train_use_spot_instances=True,        # Use spot instance\n",
    "                                    train_max_run=600,                    # Max training time\n",
    "                                    train_max_wait=3600)                  # Max training time + spot waiting time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting hyper parameters\n",
    "Each built-in algorithm has a set of hyperparameters. [Here](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost_hyperparameters.html) are the ones for XGBoost.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: how deep should the trees be?\n",
    "Tree depth is obviously an important parameter for tree-based algorithms. XGBoost has an hyper parameter named *max_depth*: by default, it is set to 6. Is this too high? Too low? Maybe we could improve our accuracy by building a more complex model based on a deeper tree. Or maybe the model would generalize better with a shallower tree?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So... Selecting these values is not obvious, is it? \n",
    "\n",
    "And what about all the other hyperparameters? **Guessing is not a strategy**. \n",
    "\n",
    "Fortunately, SageMaker supports Automatic Model Tuning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding Automatic Model Tuning\n",
    "\n",
    "We will use SageMaker tuning to automate the searching process effectively. Specifically, we specify a range, or a list of possible values in the case of categorical hyperparameters, for each of the hyperparameter that we plan to tune. SageMaker hyperparameter tuning will automatically launch multiple training jobs with different hyperparameter settings, evaluate results of those training jobs based on a predefined \"objective metric\", and select the hyperparameter settings for future attempts based on previous results. For each hyperparameter tuning job, we will give it a budget (max number of training jobs) and it will complete once that many training jobs have been executed.\n",
    "\n",
    "In this example, we are using SageMaker Python SDK to set up and manage the hyperparameter tuning job. We first configure the training jobs the hyperparameter tuning job will launch by initiating an estimator, which includes:\n",
    "* The container image for the algorithm (XGBoost)\n",
    "* Configuration for the output of the training jobs\n",
    "* The values of static algorithm hyperparameters, those that are not specified will be given default values\n",
    "* The type and number of instances to use for the training jobs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will tune four hyperparameters in this example. Don't worry if this sounds over-complicated, we don't need to understand this in detail right now.\n",
    "* *eta*: Step size shrinkage used in updates to prevent overfitting. After each boosting step, you can directly get the weights of new features. The eta parameter actually shrinks the feature weights to make the boosting process more conservative. \n",
    "* *alpha*: L1 regularization term on weights. Increasing this value makes models more conservative. \n",
    "* *min_child_weight*: Minimum sum of instance weight (hessian) needed in a child. If the tree partition step results in a leaf node with the sum of instance weight less than min_child_weight, the building process gives up further partitioning. In linear regression models, this simply corresponds to a minimum number of instances needed in each node. The larger the algorithm, the more conservative it is.\n",
    "* *max_depth*: Maximum depth of a tree. Increasing this value makes the model more complex and likely to be overfitted. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import IntegerParameter, ContinuousParameter\n",
    "\n",
    "hyperparameter_ranges = {'eta': ContinuousParameter(0, 1),\n",
    "                        'min_child_weight': ContinuousParameter(1, 10),\n",
    "                        'alpha': ContinuousParameter(0, 2),\n",
    "                        'max_depth': IntegerParameter(1, 10)\n",
    "                        }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we'll specify the objective metric that we'd like to tune for and its definition. Several metrics are [available](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost-tuning.html).\n",
    "\n",
    "As our dataset is very unbalanced, accuracy may not be the best metric here: a dumb model predicting zero all the time would be right 80%+ of the time!\n",
    "\n",
    "The [F1 score](https://en.wikipedia.org/wiki/F1_score) is a more balanced metric for classifiers, let's use that one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "objective_metric_name = 'validation:f1'\n",
    "objective_type = 'Maximize'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll create a `HyperparameterTuner` object, to which we pass:\n",
    "- The XGBoost estimator we created above\n",
    "- Our hyperparameter ranges\n",
    "- Objective metric name and definition\n",
    "- Tuning resource configurations such as Number of training jobs to run in total and how many training jobs can be run in parallel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import HyperparameterTuner\n",
    "\n",
    "xgb.set_hyperparameters(objective='binary:logistic', \n",
    "                        num_round=100,\n",
    "                        early_stopping_rounds=10)\n",
    "\n",
    "tuner = HyperparameterTuner(xgb,\n",
    "                            objective_metric_name,\n",
    "                            hyperparameter_ranges,\n",
    "                            objective_type=objective_type,\n",
    "                            max_jobs=10,\n",
    "                            max_parallel_jobs=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Launching Automatic Model Tuning\n",
    "Now we can launch a hyperparameter tuning job by calling **fit()** function. Once the job is launched, head out to SageMaker console to track the progress of the hyperparameter tuning job until it is completed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuner.fit({'train': s3_input_train, 'validation': s3_input_validation})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's just run a quick check of the tuning job status to make sure it started successfully. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker-xgboost-200204-1747\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Completed'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sagemaker = boto3.Session().client(service_name='sagemaker') \n",
    "\n",
    "# Get tuning job name\n",
    "job_name = tuner.latest_tuning_job.job_name\n",
    "print(job_name)\n",
    "\n",
    "sagemaker.describe_hyper_parameter_tuning_job(\n",
    "    HyperParameterTuningJobName=job_name)['HyperParameterTuningJobStatus']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This job will run for 10-15 minutes, so there's time for a coffee break too. If you're too hardcore for a coffee break, you can use the time to learn more about the XGBoost algo:\n",
    "* [Documentation](https://xgboost.readthedocs.io/en/latest/)\n",
    "* [Research paper](https://arxiv.org/abs/1603.02754)\n",
    "\n",
    "You can also read about the [F1 metric](https://en.wikipedia.org/wiki/F1_score) that we used: \n",
    "* **Precision**: the ratio tp / (tp + fp) where tp is the number of true positives and fp the number of false positives. The precision is intuitively the ability of the classifier not to label as positive a sample that is negative.\n",
    "* **Recall**: the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples.\n",
    "* **F1 score**: a weighted mean of precision and recall. 1 is the best possible score and 0 is the worst."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the job is running, why not head out to the SageMaker web console and spend a few minutes familiarizing yourself with the \"Hyperparameter tuning jobs\" section?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can repeatedly run the two cells below while the job is running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 training jobs have completed\n"
     ]
    }
   ],
   "source": [
    "tuning_job_result = sagemaker.describe_hyper_parameter_tuning_job(HyperParameterTuningJobName=job_name)\n",
    "\n",
    "status = tuning_job_result['HyperParameterTuningJobStatus']\n",
    "if status != 'Completed':\n",
    "    print('Reminder: the tuning job has not been completed.')\n",
    "    \n",
    "job_count = tuning_job_result['TrainingJobStatusCounters']['Completed']\n",
    "print(\"%d training jobs have completed\" % job_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best model found so far:\n",
      "'sagemaker-xgboost-200204-1747-010-7f9ee110'\n",
      "{'MetricName': 'validation:f1', 'Value': 0.647212028503418}\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint\n",
    "if tuning_job_result.get('BestTrainingJob',None):\n",
    "    print(\"Best model found so far:\")\n",
    "    pprint(tuning_job_result['BestTrainingJob']['TrainingJobName'])\n",
    "    pprint(tuning_job_result['BestTrainingJob']['FinalHyperParameterTuningJobObjectiveMetric'])\n",
    "else:\n",
    "    print(\"No training jobs have reported results yet.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect jobs with Amazon SageMaker Experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model tuning automatically creates a new experiment, and pushes information for each job. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.analytics import HyperparameterTuningJobAnalytics\n",
    "\n",
    "exp = HyperparameterTuningJobAnalytics(\n",
    "    sagemaker_session=sess, \n",
    "    hyperparameter_tuning_job_name=tuner.latest_tuning_job.name\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = exp.dataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FinalObjectiveValue</th>\n",
       "      <th>TrainingElapsedTimeSeconds</th>\n",
       "      <th>TrainingEndTime</th>\n",
       "      <th>TrainingJobName</th>\n",
       "      <th>TrainingJobStatus</th>\n",
       "      <th>TrainingStartTime</th>\n",
       "      <th>alpha</th>\n",
       "      <th>eta</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>min_child_weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.647212</td>\n",
       "      <td>61.0</td>\n",
       "      <td>2020-02-04 18:07:36+00:00</td>\n",
       "      <td>sagemaker-xgboost-200204-1747-010-7f9ee110</td>\n",
       "      <td>Completed</td>\n",
       "      <td>2020-02-04 18:06:35+00:00</td>\n",
       "      <td>0.098585</td>\n",
       "      <td>0.97199</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3.191692</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   FinalObjectiveValue  TrainingElapsedTimeSeconds           TrainingEndTime  \\\n",
       "0             0.647212                        61.0 2020-02-04 18:07:36+00:00   \n",
       "\n",
       "                              TrainingJobName TrainingJobStatus  \\\n",
       "0  sagemaker-xgboost-200204-1747-010-7f9ee110         Completed   \n",
       "\n",
       "          TrainingStartTime     alpha      eta  max_depth  min_child_weight  \n",
       "0 2020-02-04 18:06:35+00:00  0.098585  0.97199        7.0          3.191692  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values('FinalObjectiveValue', ascending=0)[:1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the tuning job is complete, we can deploy the best model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploying the best model\n",
    "\n",
    "The easiest way to deploy the best model is to use the **deploy()** API of the current [HyperparameterTuner](https://sagemaker.readthedocs.io/en/stable/tuner.html) object. If we wanted to use a previous tuning job, you could use the **attach()** API to attach to it before calling **deploy()**.\n",
    "\n",
    "The training log will be displayed. The last lines show how much you saved thanks to Managed Spot Training.\n",
    "\n",
    "While you're waiting, head out to the SageMaker web console and familiarize yourself with the \"Endpoints\" section\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker-xgboost-200204-1747-010-7f9ee110\n",
      "sagemaker-xgboost-200204-1747-010-7f9ee110-ep\n"
     ]
    }
   ],
   "source": [
    "tuning_job_result = sagemaker.describe_hyper_parameter_tuning_job(HyperParameterTuningJobName=job_name)\n",
    "best_model_name = tuning_job_result['BestTrainingJob']['TrainingJobName']\n",
    "print(best_model_name)\n",
    "\n",
    "import time\n",
    "endpoint_name = best_model_name + '-ep'\n",
    "print(endpoint_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-04 18:07:36 Starting - Preparing the instances for training\n",
      "2020-02-04 18:07:36 Downloading - Downloading input data\n",
      "2020-02-04 18:07:36 Training - Training image download completed. Training in progress.\n",
      "2020-02-04 18:07:36 Uploading - Uploading generated training model\n",
      "2020-02-04 18:07:36 Completed - Training job completed\u001b[34mINFO:sagemaker-containers:Imported framework sagemaker_xgboost_container.training\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:Failed to parse hyperparameter _tuning_objective_metric value validation:f1 to Json.\u001b[0m\n",
      "\u001b[34mReturning the value itself\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:Failed to parse hyperparameter objective value binary:logistic to Json.\u001b[0m\n",
      "\u001b[34mReturning the value itself\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34mINFO:sagemaker_xgboost_container.training:Running XGBoost Sagemaker in algorithm mode\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[18:07:25] 28831x58 matrix with 1672198 entries loaded from /opt/ml/input/data/train?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[18:07:26] 8238x58 matrix with 477804 entries loaded from /opt/ml/input/data/validation?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[34mINFO:root:Single node training.\u001b[0m\n",
      "\u001b[34mINFO:root:Setting up HPO optimized metric to be : f1\u001b[0m\n",
      "\u001b[34mINFO:root:Train matrix has 28831 rows\u001b[0m\n",
      "\u001b[34mINFO:root:Validation matrix has 8238 rows\u001b[0m\n",
      "\u001b[34m[0]#011train-error:0.098228#011validation-error:0.10136#011train-f1:0.645224#011validation-f1:0.623886\u001b[0m\n",
      "\u001b[34m[1]#011train-error:0.097707#011validation-error:0.102816#011train-f1:0.653369#011validation-f1:0.621394\u001b[0m\n",
      "\u001b[34m[2]#011train-error:0.097118#011validation-error:0.102573#011train-f1:0.657956#011validation-f1:0.627399\u001b[0m\n",
      "\u001b[34m[3]#011train-error:0.096355#011validation-error:0.10136#011train-f1:0.662872#011validation-f1:0.63236\u001b[0m\n",
      "\u001b[34m[4]#011train-error:0.094933#011validation-error:0.102331#011train-f1:0.674706#011validation-f1:0.636451\u001b[0m\n",
      "\u001b[34m[5]#011train-error:0.093615#011validation-error:0.102573#011train-f1:0.678321#011validation-f1:0.636651\u001b[0m\n",
      "\u001b[34m[6]#011train-error:0.093337#011validation-error:0.102331#011train-f1:0.680556#011validation-f1:0.641683\u001b[0m\n",
      "\u001b[34m[7]#011train-error:0.093337#011validation-error:0.10318#011train-f1:0.685819#011validation-f1:0.644572\u001b[0m\n",
      "\u001b[34m[8]#011train-error:0.091811#011validation-error:0.10318#011train-f1:0.69168#011validation-f1:0.647549\u001b[0m\n",
      "\u001b[34m[9]#011train-error:0.091429#011validation-error:0.103302#011train-f1:0.692901#011validation-f1:0.64836\u001b[0m\n",
      "\u001b[34m[10]#011train-error:0.090493#011validation-error:0.104394#011train-f1:0.697875#011validation-f1:0.645871\u001b[0m\n",
      "\u001b[34m[11]#011train-error:0.089869#011validation-error:0.103423#011train-f1:0.700886#011validation-f1:0.647212\u001b[0m\n",
      "Training seconds: 61\n",
      "Billable seconds: 21\n",
      "Managed Spot Training savings: 65.6%\n",
      "---------------------!"
     ]
    }
   ],
   "source": [
    "xgb_endpoint = tuner.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge', endpoint_name=endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting with our model\n",
    "\n",
    "First we'll need to determine how we pass data into and receive data from our endpoint. Our data is currently stored as NumPy arrays in memory of our notebook instance. To send it in an HTTP POST request, we'll serialize it as a CSV string and then decode the resulting CSV.\n",
    "\n",
    "Now, we'll use a simple function to:\n",
    "* Loop over our test dataset\n",
    "* Split it into mini-batches of rows\n",
    "* Convert those mini-batches to CSV string payloads (of course, we drop the target variable)\n",
    "* Retrieve mini-batch predictions by invoking the XGBoost endpoint\n",
    "* Collect predictions and convert from the CSV output our model provides into a NumPy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.array_split.html\n",
    "\n",
    "def predict(data, rows=500):\n",
    "    split_array = np.array_split(data, int(data.shape[0] / float(rows) + 1))\n",
    "    predictions = ''\n",
    "    for array in split_array:\n",
    "        predictions = ','.join([predictions, xgb_endpoint.predict(array).decode('utf-8')])\n",
    "\n",
    "    return np.fromstring(predictions[1:], sep=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.09145568 0.10820587 0.10820587 ... 0.10820587 0.18167706 0.10820587]\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.predictor import csv_serializer\n",
    "\n",
    "xgb_endpoint.content_type = 'text/csv'\n",
    "xgb_endpoint.serializer = csv_serializer\n",
    "\n",
    "# We need to drop the target value, as we're predicting it :)\n",
    "predictions = predict(test_data.drop(['y_no', 'y_yes'], axis=1).values)\n",
    "print(predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each sample, our binary classifier returns a probability between 0 and 1. Since we decided to maximize accuracy, the model sets a threshold of 0.5: anything lower is treated as a 0, anything higher as a 1. \n",
    "\n",
    "To dive a little deeper:  the threshold is baked in the metric that XGBoost uses. Here, we use the default 'eval_metric' for classification, i.e. 'error'. This metric has a default threshold of 0.5. If you look at the [XGBoost doc](https://xgboost.readthedocs.io/en/latest/parameter.html), you'll see that it's possible to pass a different threshold, doing something like:\n",
    "xgb.set_hyperparameters(objective='binary:logistic', num_round=100, eval_metric='error\\@0.2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>predictions</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>truth</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3581</td>\n",
       "      <td>57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>373</td>\n",
       "      <td>108</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "predictions     0    1\n",
       "truth                 \n",
       "0            3581   57\n",
       "1             373  108"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# https://pandas.pydata.org/pandas-docs/version/0.25/generated/pandas.crosstab.html\n",
    "\n",
    "threshold = 0.5\n",
    "rounded_predictions = np.where(predictions > threshold, 1, 0)\n",
    "\n",
    "# Also called a 'confusion matrix'\n",
    "pd.crosstab(index=test_data['y_yes'], \n",
    "            columns=rounded_predictions,\n",
    "            rownames=['truth'], colnames=['predictions'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How well did we do on the test set (your own numbers might vary):\n",
    "* 3583 true negatives and 91 true positives were correctly predicted.\n",
    "* 390 positives were incorrectly predicted as negatives (false negatives), so we'll probably miss business opportunities by not engaging with these customers. It looks like this model is too conservative!\n",
    "* 55 negatives were incorrectly predicted as positives (false positives), so we'll probably waste our time engaging with these customers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keep in mind that you can either optimize false positives or false negatives, but not both. You have to decide which ones have the bigger impact on your application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This trade-off is made more difficult by the class imbalance problem. Fixing it is beyond the scope of this workshop, but we could use techniques like:\n",
    "* adding real data to the positive class (real or synthetic data),\n",
    "* adding synthetic data to the positive class (e.g. over-sampling),\n",
    "* using the 'scale_pos_weight' hyper parameter to account for class imbalance,\n",
    "* more pre-processing, more feature engineering, etc.\n",
    "\n",
    "If you want to dive deeper, these blog posts are a good starting point: \n",
    "* https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/\n",
    "* http://francescopochetti.com/extreme-label-imbalance-when-you-measure-the-minority-class-in-basis-points/ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deleting the endpoint\n",
    "Once that we're done predicting, we can delete the endpoint (and stop paying for it). You can re-deploy again by running the appropriate cell above. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we're done, we can delete the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ResponseMetadata': {'RequestId': '7c8c4be8-07e9-4f7b-ac0e-103b73d2861d',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '7c8c4be8-07e9-4f7b-ac0e-103b73d2861d',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '0',\n",
       "   'date': 'Tue, 04 Feb 2020 18:42:10 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sagemaker.delete_endpoint(EndpointName=endpoint_name)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
